Understanding the Chrome Sync Protocol
Chrome is a cool browser, but its secret sauce is that no matter whether you are using iOS, Windows, Mac, Android, Linux or ChromeOS, you can sync your bookmarks, passwords, recently viewed URLs and more.
Did you noticed any OS missing? No? OK, so perhaps you don’t use Windows Phone.
But I do, as well as Android and iOS, and it bugged me that there was no way to sync all my Chrome goodness to Windows Phone, since Chrome is not available for Windows Phone.
So I implemented my own Chrome sync engine on Windows Phone, and in the process learned how Chrome sync works.
In this post I'll share what I learned, including how you authenticate in order to use it.
I'm going to do this by way of the free Chrome sync app I created for Windows Phone, called Chrync.
I reasoned that there must be a way of talking the Chrome sync protocol directly to Google's servers, since Chrome itself does it.
I started off by downloading the Chrome source code, and building it, and running it with a debugger attached.
I also discovered the wonderful world of Chrome debug pages, which are very helpful, especially the sync internals page which you can access by navigating to chrome://sync-internals/
Protocol Buffers
I found that the Chrome sync protocol is layered on top of a Google technology called Protocol Buffers, with the Chrome sync structures being defined in a language independent protocol buffers IDL.
The main source is at http://src.chromium.org/viewvc/chrome/trunk/src/sync/protocol/, and there you’ll find the message types that are sent to and from the Google servers when a sync occurs.
If you want to browse, I suggest starting with sync.proto which defines the SyncEntity message containing core sync item fields, including an EntitySpecifics (also defined in sync.proto).
The EntitySpecifics message contains a load of optional fields such as BookmarkSpecifics (used for syncing bookmarks), TypedUrlSpecifics (recently browsed URLs), PasswordSpecifics (saved passwords), SessionSpecifics (open sessions) and NigoriSpecifics decrypting all this stuff).
Over time various extensions have been defined. Indeed every time I check the GIT source repository it seems that something new is happening, such as SyncedNotificationSpecifics.
Converting the protocol definitions to native code
I wanted to talk the Chrome protocol on Windows Phone, and went hunting for a C# implementation of Protocol Buffers that worked on Windows Phone. I found two: protobuf-net by Marc Gravell and protobuf-csharp-port by Jon Skeet which I ended up using.
I was able to generate C# proxies for the Chrome sync protocol buffer files, and link in the .NET protocol buffers runtime.
The next step was to work out how to authenticate.
Requesting OAuth 2.0 access to Chrome sync data
Like many Google users, I use two factor authentication, and since I am especially paranoid, I have a custom Chrome sync passphrase defined.
Since I was making the app mainly for myself I needed to support both two factor authentication and custom passphrases.
Google has a standard OAuth 2.0 implementation which they describe here.
You direct the user to a Google web site with an authentication request to Google, specifying in the scope parameter what access you require, for example you use userinfo.email to request access to the user’s email address.
You can indicate that your app requires access to all kinds of Google services using the Google Cloud Console. You’ll notice though that there way to specify to access a user’s Chrome sync data.
After a little digging I discovered the magic string to request access in the scope parameter to Chrome sync data. In fact I ask for access to the user’s email address, and their Chrome sync data. The scope I use is https://www.googleapis.com/auth/userinfo.email+https://www.googleapis.com/auth/chromesync
Below you see the OAuth 2.0 process in progress inside a web browser I host within the app. You login, using two factor authentication if it is enabled, and then you get prompted to ask whether you want to give the app the access that it requests.
For some reason, Google’s OAuth prompts are always in German for me, despite the fact that I speak no German, and although I live in Switzerland, I live in a French speaking area. If you don’t speak German you’ll have to take my word for it that it is prompting for permission to access your email address and your Chrome sync data.
The result of this authentication are two tokens: an access token, which is good for a certain amount of time, and a refresh token, which can be used to generate a new access token when it expires.
Building the sync request
Initiating the sync process involves making an http request to https://clients4.google.com/chrome-sync and setting a “Bearer” http header to the access token. The body of the message is an octet-stream which contains the sync request.
The sync request itself is a GetUpdatesMessage defined in a ClientToServerMessage which are defined in sync.proto:
GetUpdatesMessage
message GetUpdatesMessage {
// Indicates the client's current progress in downloading updates. A
// from_timestamp value of zero means that the client is requesting a first-
// time sync. After that point, clients should fill in this value with the
// value returned in the last-seen GetUpdatesResponse.new_timestamp.
//
// from_timestamp has been deprecated; clients should use
// |from_progress_marker| instead, which allows more flexibility.
optional int64 from_timestamp = 1;
// Indicates the reason for the GetUpdatesMessage.
// Deprecated in M29. We should eventually rely on GetUpdatesOrigin instead.
// Newer clients will support both systems during the transition period.
optional GetUpdatesCallerInfo caller_info = 2;
// Indicates whether related folders should be fetched.
optional bool fetch_folders = 3 [default = true];
// The presence of an individual EntitySpecifics field indicates that the
// client requests sync object types associated with that field. This
// determination depends only on the presence of the field, not its
// contents -- thus clients should send empty messages as the field value.
// For backwards compatibility only bookmark objects will be sent to the
// client should requested_types not be present.
//
// requested_types may contain multiple EntitySpecifics fields -- in this
// event, the server will return items of all the indicated types.
//
// requested_types has been deprecated; clients should use
// |from_progress_marker| instead, which allows more flexibility.
optional EntitySpecifics requested_types = 4;
// Client-requested limit on the maximum number of updates to return at once.
// The server may opt to return fewer updates than this amount, but it should
// not return more.
optional int32 batch_size = 5;
// Per-datatype progress marker. If present, the server will ignore
// the values of requested_types and from_timestamp, using this instead.
//
// With the exception of certain configuration or initial sync requests, the
// client should include one instance of this field for each enabled data
// type.
repeated DataTypeProgressMarker from_progress_marker = 6;
// Indicates whether the response should be sent in chunks. This may be
// needed for devices with limited memory resources. If true, the response
// will include one or more ClientToServerResponses, with the frist one
// containing GetUpdatesMetadataResponse, and the remaining ones, if any,
// containing GetUpdatesStreamingResponse. These ClientToServerResponses are
// delimited by a length prefix, which is encoded as a varint.
optional bool streaming = 7 [default = false];
// Whether the client needs the server to provide an encryption key for this
// account.
// Note: this should typically only be set on the first GetUpdates a client
// requests. Clients are expected to persist the encryption key from then on.
// The allowed frequency for requesting encryption keys is much lower than
// other datatypes, so repeated usage will likely result in throttling.
optional bool need_encryption_key = 8 [default = false];
// Whether to create the mobile bookmarks folder if it's not
// already created. Should be set to true only by mobile clients.
optional bool create_mobile_bookmarks_folder = 1000 [default = false];
// This value is an updated version of the GetUpdatesCallerInfo's
// GetUpdatesSource. It describes the reason for the GetUpdate request.
// Introduced in M29.
optional SyncEnums.GetUpdatesOrigin get_updates_origin = 9;
// Whether this GU also serves as a retry GU. Any GU that happens after
// retry timer timeout is a retry GU effectively.
optional bool is_retry = 10 [default = false];
};
This is my code to build this sync request:
/// <summary>
/// Builds a sync request to be sent to the server. Initializes it based on the user's selected
/// sync options, and previous sync state
/// </summary>
/// <returns></returns>
private byte[] BuildSyncRequest() {
D("BuildSyncRequest invoked");
// This ClientToServerMessage is generated from the sync.proto definition
var myRequest = ClientToServerMessage.CreateBuilder();
myRequest.SetShare(_syncOptions.User);
using (var db = _databaseFactory.Get()) {
if (db == null) throw new Exception("User logged out");
var syncState = db.GetSyncState();
// We want to get updates, other options include COMMIT to send changes
myRequest.SetMessageContents(ClientToServerMessage.Types.Contents.GET_UPDATES);
var callerInfo = GetUpdatesCallerInfo.CreateBuilder();
callerInfo.NotificationsEnabled = true;
callerInfo.SetSource(GetUpdatesCallerInfo.Types.GetUpdatesSource.PERIODIC);
var getUpdates = GetUpdatesMessage.CreateBuilder();
getUpdates.SetCallerInfo(callerInfo);
getUpdates.SetFetchFolders(true);
// Tell the server what kinds of sync items we can handle
// We need this in case the user has encrypted everything ... nigori is to get decryption
// keys to decrypted encrypted items
var nigoriDataType = InitializeDataType(db, EntitySpecifics.NigoriFieldNumber);
getUpdates.FromProgressMarkerList.Add(nigoriDataType.Build());
// We include bookmarks if the user selected them
if ((_syncOptions.Flags & SyncFlags.Bookmarks) == SyncFlags.Bookmarks) {
// The field is initialized with state information from the last sync, if any, so that
// we only get changes since the latest sync
var bookmarkDataType = InitializeDataType(db, EntitySpecifics.BookmarkFieldNumber);
getUpdates.FromProgressMarkerList.Add(bookmarkDataType.Build());
}
if ((_syncOptions.Flags & SyncFlags.OpenTabs) == SyncFlags.OpenTabs) {
var sessionDataType = InitializeDataType(db, EntitySpecifics.SessionFieldNumber);
getUpdates.FromProgressMarkerList.Add(sessionDataType.Build());
}
if ((_syncOptions.Flags & SyncFlags.Omnibox) == SyncFlags.Omnibox) {
var typedUrlDataType = InitializeDataType(db, EntitySpecifics.TypedUrlFieldNumber);
getUpdates.FromProgressMarkerList.Add(typedUrlDataType.Build());
}
if ((_syncOptions.Flags & SyncFlags.Passwords) == SyncFlags.Passwords) {
var passwordDataType = InitializeDataType(db, EntitySpecifics.PasswordFieldNumber);
getUpdates.FromProgressMarkerList.Add(passwordDataType.Build());
}
if (syncState != null) {
// ChipBag is "Per-client state for use by the server. Sent with every message sent to the server."
// Soggy newspaper not included
if (syncState.ChipBag != null) {
var chipBag = ChipBag.CreateBuilder().SetServerChips(ByteString.CopyFrom(syncState.ChipBag)).Build();
myRequest.SetBagOfChips(chipBag);
}
if (syncState.StoreBirthday != null) {
myRequest.SetStoreBirthday(syncState.StoreBirthday);
}
}
myRequest.SetGetUpdates(getUpdates);
myRequest.SetClientStatus(ClientStatus.CreateBuilder().Build());
}
var builtRequest = myRequest.Build();
return builtRequest.ToByteArray();
}
/// <summary>
/// For each item type we sync, this method initializes it
/// </summary>
private DataTypeProgressMarker.Builder InitializeDataType(IDatabase db, int fieldNumber) {
var dataType = DataTypeProgressMarker.CreateBuilder();
dataType.SetDataTypeId(fieldNumber);
InitializeMarker(dataType, db);
return dataType;
}
/// <summary>
/// Initializes the sync state for the item types we sync
/// </summary>
private void InitializeMarker(DataTypeProgressMarker.Builder dataType, IDatabase db) {
var marker = db.GetSyncProgress(dataType.DataTypeId);
if (marker == null) {
return;
}
D("Initializing marker: " + marker);
if (marker.NotificationHint != null) {
dataType.SetNotificationHint(marker.NotificationHint);
}
dataType.SetToken(ByteString.CopyFrom(marker.Token));
if (marker.TimestampForMigration != 0) {
dataType.SetTimestampTokenForMigration(marker.TimestampForMigration);
}
}
Handling the sync response
Once this request is sent off we get back a sync response, in the form of a ClientToServerResponse containing a GetUpdatesResponse, which are also defined in sync.proto:
GetUpdatesResponse
message GetUpdatesResponse {
// New sync entries that the client should apply.
repeated SyncEntity entries = 1;
// If there are more changes on the server that weren't processed during this
// GetUpdates request, the client should send another GetUpdates request and
// use new_timestamp as the from_timestamp value within GetUpdatesMessage.
//
// This field has been deprecated and will be returned only to clients
// that set the also-deprecated |from_timestamp| field in the update request.
// Clients should use |from_progress_marker| and |new_progress_marker|
// instead.
optional int64 new_timestamp = 2;
// DEPRECATED FIELD - server does not set this anymore.
optional int64 deprecated_newest_timestamp = 3;
// Approximate count of changes remaining - use this for UI feedback.
// If present and zero, this estimate is firm: the server has no changes
// after the current batch.
optional int64 changes_remaining = 4;
// Opaque, per-datatype timestamp-like tokens. A client should use this
// field in lieu of new_timestamp, which is deprecated in newer versions
// of the protocol. Clients should retain and persist the values returned
// in this field, and present them back to the server to indicate the
// starting point for future update requests.
//
// This will be sent only if the client provided |from_progress_marker|
// in the update request.
//
// The server may provide a new progress marker even if this is the end of
// the batch, or if there were no new updates on the server; and the client
// must save these. If the server does not provide a |new_progress_marker|
// value for a particular datatype, when the request provided a
// |from_progress_marker| value for that datatype, the client should
// interpret this to mean "no change from the previous state" and retain its
// previous progress-marker value for that datatype.
//
// Progress markers in the context of a response will never have the
// |timestamp_token_for_migration| field set.
repeated DataTypeProgressMarker new_progress_marker = 5;
// The current encryption keys associated with this account. Will be set if
// the GetUpdatesMessage in the request had need_encryption_key == true or
// the server has updated the set of encryption keys (e.g. due to a key
// rotation).
repeated bytes encryption_keys = 6;
};
SyncEntity
Note that at the start of GetUpdatesResponse there is a repeated series of SyncEntities. SyncEntity is also defined in sync.proto:
message SyncEntity {
// This item's identifier. In a commit of a new item, this will be a
// client-generated ID. If the commit succeeds, the server will generate
// a globally unique ID and return it to the committing client in the
// CommitResponse.EntryResponse. In the context of a GetUpdatesResponse,
// |id_string| is always the server generated ID. The original
// client-generated ID is preserved in the |originator_client_id| field.
// Present in both GetUpdatesResponse and CommitMessage.
optional string id_string = 1;
// An id referencing this item's parent in the hierarchy. In a
// CommitMessage, it is accepted for this to be a client-generated temporary
// ID if there was a new created item with that ID appearing earlier
// in the message. In all other situations, it is a server ID.
// Present in both GetUpdatesResponse and CommitMessage.
optional string parent_id_string = 2;
// old_parent_id is only set in commits and indicates the old server
// parent(s) to remove. When omitted, the old parent is the same as
// the new.
// Present only in CommitMessage.
optional string old_parent_id = 3;
// The version of this item -- a monotonically increasing value that is
// maintained by for each item. If zero in a CommitMessage, the server
// will interpret this entity as a newly-created item and generate a
// new server ID and an initial version number. If nonzero in a
// CommitMessage, this item is treated as an update to an existing item, and
// the server will use |id_string| to locate the item. Then, if the item's
// current version on the server does not match |version|, the commit will
// fail for that item. The server will not update it, and will return
// a result code of CONFLICT. In a GetUpdatesResponse, |version| is
// always positive and indentifies the revision of the item data being sent
// to the client.
// Present in both GetUpdatesResponse and CommitMessage.
required int64 version = 4;
// Last modification time (in java time milliseconds)
// Present in both GetUpdatesResponse and CommitMessage.
optional int64 mtime = 5;
// Creation time.
// Present in both GetUpdatesResponse and CommitMessage.
optional int64 ctime = 6;
// The name of this item.
// Historical note:
// Since November 2010, this value is no different from non_unique_name.
// Before then, server implementations would maintain a unique-within-parent
// value separate from its base, "non-unique" value. Clients had not
// depended on the uniqueness of the property since November 2009; it was
// removed from Chromium by http://codereview.chromium.org/371029 .
// Present in both GetUpdatesResponse and CommitMessage.
required string name = 7;
// The name of this item. Same as |name|.
// |non_unique_name| should take precedence over the |name| value if both
// are supplied. For efficiency, clients and servers should avoid setting
// this redundant value.
// Present in both GetUpdatesResponse and CommitMessage.
optional string non_unique_name = 8;
// A value from a monotonically increasing sequence that indicates when
// this item was last updated on the server. This is now equivalent
// to version. This is now deprecated in favor of version.
// Present only in GetUpdatesResponse.
optional int64 sync_timestamp = 9;
// If present, this tag identifies this item as being a uniquely
// instanced item. The server ensures that there is never more
// than one entity in a user's store with the same tag value.
// This value is used to identify and find e.g. the "Google Chrome" settings
// folder without relying on it existing at a particular path, or having
// a particular name, in the data store.
//
// This variant of the tag is created by the server, so clients can't create
// an item with a tag using this field.
//
// Use client_defined_unique_tag if you want to create one from the client.
//
// An item can't have both a client_defined_unique_tag and
// a server_defined_unique_tag.
//
// Present only in GetUpdatesResponse.
optional string server_defined_unique_tag = 10;
// If this group is present, it implies that this SyncEntity corresponds to
// a bookmark or a bookmark folder.
//
// This group is deprecated; clients should use the bookmark EntitySpecifics
// protocol buffer extension instead.
optional group BookmarkData = 11 {
// We use a required field to differentiate between a bookmark and a
// bookmark folder.
// Present in both GetUpdatesMessage and CommitMessage.
required bool bookmark_folder = 12;
// For bookmark objects, contains the bookmark's URL.
// Present in both GetUpdatesResponse and CommitMessage.
optional string bookmark_url = 13;
// For bookmark objects, contains the bookmark's favicon. The favicon is
// represented as a 16X16 PNG image.
// Present in both GetUpdatesResponse and CommitMessage.
optional bytes bookmark_favicon = 14;
}
// Supplies a numeric position for this item, relative to other items with the
// same parent. Deprecated in M26, though clients are still required to set
// it.
//
// Present in both GetUpdatesResponse and CommitMessage.
//
// At one point this was used as an alternative / supplement to
// the deprecated |insert_after_item_id|, but now it, too, has been
// deprecated.
//
// In order to maintain compatibility with older clients, newer clients should
// still set this field. Its value should be based on the first 8 bytes of
// this item's |unique_position|.
//
// Nerwer clients must also support the receipt of items that contain
// |position_in_parent| but no |unique_position|. They should locally convert
// the given int64 position to a UniquePosition.
//
// The conversion from int64 to UniquePosition is as follows:
// The int64 value will have its sign bit flipped then placed in big endian
// order as the first 8 bytes of the UniquePosition. The subsequent bytes of
// the UniquePosition will consist of the item's unique suffix.
//
// Conversion from UniquePosition to int64 reverses this process: the first 8
// bytes of the position are to be interpreted as a big endian int64 value
// with its sign bit flipped.
optional int64 position_in_parent = 15;
// Contains the ID of the element (under the same parent) after which this
// element resides. An empty string indicates that the element is the first
// element in the parent. This value is used during commits to specify
// a relative position for a position change. In the context of
// a GetUpdatesMessage, |position_in_parent| is used instead to
// communicate position.
//
// Present only in CommitMessage.
//
// This is deprecated. Clients are allowed to omit this as long as they
// include |position_in_parent| instead.
optional string insert_after_item_id = 16;
// Arbitrary key/value pairs associated with this item.
// Present in both GetUpdatesResponse and CommitMessage.
// Deprecated.
// optional ExtendedAttributes extended_attributes = 17;
// If true, indicates that this item has been (or should be) deleted.
// Present in both GetUpdatesResponse and CommitMessage.
optional bool deleted = 18 [default = false];
// A GUID that identifies the the sync client who initially committed
// this entity. This value corresponds to |cache_guid| in CommitMessage.
// This field, along with |originator_client_item_id|, can be used to
// reunite the original with its official committed version in the case
// where a client does not receive or process the commit response for
// some reason.
//
// Present only in GetUpdatesResponse.
//
// This field is also used in determining the unique identifier used in
// bookmarks' unique_position field.
optional string originator_cache_guid = 19;
// The local item id of this entry from the client that initially
// committed this entity. Typically a negative integer.
// Present only in GetUpdatesResponse.
//
// This field is also used in determinging the unique identifier used in
// bookmarks' unique_position field.
optional string originator_client_item_id = 20;
// Extensible container for datatype-specific data.
// This became available in version 23 of the protocol.
optional EntitySpecifics specifics = 21;
// Indicate whether this is a folder or not. Available in version 23+.
optional bool folder = 22 [default = false];
// A client defined unique hash for this entity.
// Similar to server_defined_unique_tag.
//
// When initially committing an entity, a client can request that the entity
// is unique per that account. To do so, the client should specify a
// client_defined_unique_tag. At most one entity per tag value may exist.
// per account. The server will enforce uniqueness on this tag
// and fail attempts to create duplicates of this tag.
// Will be returned in any updates for this entity.
//
// The difference between server_defined_unique_tag and
// client_defined_unique_tag is the creator of the entity. Server defined
// tags are entities created by the server at account creation,
// while client defined tags are entities created by the client at any time.
//
// During GetUpdates, a sync entity update will come back with ONE of:
// a) Originator and cache id - If client committed the item as non "unique"
// b) Server tag - If server committed the item as unique
// c) Client tag - If client committed the item as unique
//
// May be present in CommitMessages for the initial creation of an entity.
// If present in Commit updates for the entity, it will be ignored.
//
// Available in version 24+.
//
// May be returned in GetUpdatesMessage and sent up in CommitMessage.
//
optional string client_defined_unique_tag = 23;
// This positioning system had a relatively short life. It was made obsolete
// by |unique_position| before either the client or server made much of an
// attempt to support it. In fact, no client ever read or set this field.
//
// Deprecated in M26.
optional bytes ordinal_in_parent = 24;
// This is the fourth attempt at positioning.
//
// This field is present in both GetUpdatesResponse and CommitMessage, if the
// item's type requires it and the client that wrote the item supports it (M26
// or higher). Clients must also be prepared to handle updates from clients
// that do not set this field. See the comments on
// |server_position_in_parent| for more information on how this is handled.
//
// This field will not be set for items whose type ignores positioning.
// Clients should not attempt to read this field on the receipt of an item of
// a type that ignores positioning.
//
// Refer to its definition in unique_position.proto for more information about
// its internal representation.
optional UniquePosition unique_position = 25;
};
EntitySpecifics
What is most important in the SyncEntity is that a SyncEntity contains an EntitySpecifics, which is where the good stuff is. The EntitySpecifics looks like this:
message EntitySpecifics {
// If a datatype is encrypted, this field will contain the encrypted
// original EntitySpecifics. The extension for the datatype will continue
// to exist, but contain only the default values.
// Note that currently passwords employ their own legacy encryption scheme and
// do not use this field.
optional EncryptedData encrypted = 1;
// To add new datatype-specific fields to the protocol, extend
// EntitySpecifics. First, pick a non-colliding tag number by
// picking a revision number of one of your past commits
// to src.chromium.org. Then, in a different protocol buffer
// definition, define your message type, and add an optional field
// to the list below using the unique tag value you selected.
//
// optional MyDatatypeSpecifics my_datatype = 32222;
//
// where:
// - 32222 is the non-colliding tag number you picked earlier.
// - MyDatatypeSpecifics is the type (probably a message type defined
// in your new .proto file) that you want to associate with each
// object of the new datatype.
// - my_datatype is the field identifier you'll use to access the
// datatype specifics from the code.
//
// Server implementations are obligated to preserve the contents of
// EntitySpecifics when it contains unrecognized fields. In this
// way, it is possible to add new datatype fields without having
// to update the server.
//
// Note: The tag selection process is based on legacy versions of the
// protocol which used protobuf extensions. We have kept the process
// consistent as the old values cannot change. The 5+ digit nature of the
// tags also makes them recognizable (individually and collectively) from
// noise in logs and debugging contexts, and creating a divergent subset of
// tags would only make things a bit more confusing.
optional AutofillSpecifics autofill = 31729;
optional BookmarkSpecifics bookmark = 32904;
optional PreferenceSpecifics preference = 37702;
optional TypedUrlSpecifics typed_url = 40781;
optional ThemeSpecifics theme = 41210;
optional AppNotification app_notification = 45184;
optional PasswordSpecifics password = 45873;
optional NigoriSpecifics nigori = 47745;
optional ExtensionSpecifics extension = 48119;
optional AppSpecifics app = 48364;
optional SessionSpecifics session = 50119;
optional AutofillProfileSpecifics autofill_profile = 63951;
optional SearchEngineSpecifics search_engine = 88610;
optional ExtensionSettingSpecifics extension_setting = 96159;
optional AppSettingSpecifics app_setting = 103656;
optional HistoryDeleteDirectiveSpecifics history_delete_directive = 150251;
optional SyncedNotificationSpecifics synced_notification = 153108;
optional SyncedNotificationAppInfoSpecifics synced_notification_app_info =
235816;
optional DeviceInfoSpecifics device_info = 154522;
optional ExperimentsSpecifics experiments = 161496;
optional PriorityPreferenceSpecifics priority_preference = 163425;
optional DictionarySpecifics dictionary = 170540;
optional FaviconTrackingSpecifics favicon_tracking = 181534;
optional FaviconImageSpecifics favicon_image = 182019;
optional ManagedUserSettingSpecifics managed_user_setting = 186662;
optional ManagedUserSpecifics managed_user = 194582;
optional ManagedUserSharedSettingSpecifics managed_user_shared_setting =
202026;
optional ArticleSpecifics article = 223759;
optional AppListSpecifics app_list = 229170;
}
BookmarkSpecifics
As you see the EntitySpecifics contains EncryptedData and optional fields for each of the data types. A specific instance of an EntitySpecifics contains just one, for example here is the BookmarkSpecifics from bookmarks_specifics.proto
// Properties of bookmark sync objects.
message BookmarkSpecifics {
optional string url = 1;
optional bytes favicon = 2;
optional string title = 3;
// Corresponds to BookmarkNode::date_added() and is the internal value from
// base::Time.
optional int64 creation_time_us = 4;
optional string icon_url = 5;
repeated MetaInfo meta_info = 6;
}
Decrypting sync data
What makes things tricky is that you get a set of sync entities, some of which may be encrypted (in the EncryptedData EntitySpecifics field), but they cannot be decrypted until the NigoriSpecifics sync entity is received, which may be some time. So I buffer of the encrypted sync entities until they can be decrypted.
Encrypted data looks like this in its Protocol Buffers definition in encryption.proto:
EncryptedData
// Encrypted sync data consists of two parts: a key name and a blob. Key name is
// the name of the key that was used to encrypt blob and blob is encrypted data
// itself.
//
// The reason we need to keep track of the key name is that a sync user can
// change their passphrase (and thus their encryption key) at any time. When
// that happens, we make a best effort to reencrypt all nodes with the new
// passphrase, but since we don't have transactions on the server-side, we
// cannot guarantee that every node will be reencrypted. As a workaround, we
// keep track of all keys, assign each key a name (by using that key to encrypt
// a well known string) and keep track of which key was used to encrypt each
// node.
message EncryptedData {
optional string key_name = 1;
optional string blob = 2;
};
NigoriKey, NigoriKeyBag and NigoriSpecific
The NigoriSpecifics (one of the entries in the EntitySpecifics) looks like this, including associated data types, in nigori_specifics.proto
message NigoriKey {
optional string name = 1;
optional bytes user_key = 2;
optional bytes encryption_key = 3;
optional bytes mac_key = 4;
}
message NigoriKeyBag {
repeated NigoriKey key = 2;
}
// Properties of nigori sync object.
message NigoriSpecifics {
optional EncryptedData encryption_keybag = 1;
// Once keystore migration is performed, we have to freeze the keybag so that
// older clients (that don't support keystore encryption) do not attempt to
// update the keybag.
// Previously |using_explicit_passphrase|.
optional bool keybag_is_frozen = 2;
// Obsolete encryption fields. These were deprecated due to legacy versions
// that understand their usage but did not perform encryption properly.
// optional bool deprecated_encrypt_bookmarks = 3;
// optional bool deprecated_encrypt_preferences = 4;
// optional bool deprecated_encrypt_autofill_profile = 5;
// optional bool deprecated_encrypt_autofill = 6;
// optional bool deprecated_encrypt_themes = 7;
// optional bool deprecated_encrypt_typed_urls = 8;
// optional bool deprecated_encrypt_extensions = 9;
// optional bool deprecated_encrypt_sessions = 10;
// optional bool deprecated_encrypt_apps = 11;
// optional bool deprecated_encrypt_search_engines = 12;
// Booleans corresponding to whether a datatype should be encrypted.
// Passwords are always encrypted, so we don't need a field here.
// History delete directives need to be consumable by the server, and
// thus can't be encrypted.
// Synced Notifications need to be consumed by the server (the read flag)
// and thus can't be encrypted.
// Synced Notification App Info is set by the server, and thus cannot be
// encrypted.
optional bool encrypt_bookmarks = 13;
optional bool encrypt_preferences = 14;
optional bool encrypt_autofill_profile = 15;
optional bool encrypt_autofill = 16;
optional bool encrypt_themes = 17;
optional bool encrypt_typed_urls = 18;
optional bool encrypt_extensions = 19;
optional bool encrypt_sessions = 20;
optional bool encrypt_apps = 21;
optional bool encrypt_search_engines = 22;
// Deprecated on clients where tab sync is enabled by default.
// optional bool sync_tabs = 23;
// If true, all current and future datatypes will be encrypted.
optional bool encrypt_everything = 24;
optional bool encrypt_extension_settings = 25;
optional bool encrypt_app_notifications = 26;
optional bool encrypt_app_settings = 27;
// User device information. Contains information about each device that has a
// sync-enabled Chrome browser connected to the user account.
// This has been moved to the DeviceInfo message.
// repeated DeviceInformation deprecated_device_information = 28;
// Enable syncing favicons as part of tab sync.
optional bool sync_tab_favicons = 29;
// The state of the passphrase required to decrypt |encryption_keybag|.
enum PassphraseType {
// Gaia-based encryption passphrase. Deprecated.
IMPLICIT_PASSPHRASE = 1;
// Keystore key encryption passphrase. Uses |keystore_bootstrap| to
// decrypt |encryption_keybag|.
KEYSTORE_PASSPHRASE = 2;
// Previous Gaia-based passphrase frozen and treated as a custom passphrase.
FROZEN_IMPLICIT_PASSPHRASE = 3;
// User provided custom passphrase.
CUSTOM_PASSPHRASE = 4;
}
optional PassphraseType passphrase_type = 30
[default = IMPLICIT_PASSPHRASE];
// The keystore decryptor token blob. Encrypted with the keystore key, and
// contains the encryption key used to decrypt |encryption_keybag|.
// Only set if passphrase_state == KEYSTORE_PASSPHRASE.
optional EncryptedData keystore_decryptor_token = 31;
// The time (in epoch milliseconds) at which the keystore migration was
// performed.
optional int64 keystore_migration_time = 32;
// The time (in epoch milliseconds) at which a custom passphrase was set.
// Note: this field may not be set if the custom passphrase was applied before
// this field was introduced.
optional int64 custom_passphrase_time = 33;
// Boolean corresponding to whether custom spelling dictionary should be
// encrypted.
optional bool encrypt_dictionary = 34;
// Boolean corresponding to Whether to encrypt favicons data or not.
optional bool encrypt_favicon_images = 35;
optional bool encrypt_favicon_tracking = 36;
// Boolean corresponding to whether articles should be encrypted.
optional bool encrypt_articles = 37;
// Boolean corresponding to whether app list items should be encrypted.
optional bool encrypt_app_list = 38;
}
Note that the first item in the NigiriSpecifics is the encrypted NigoriKeyBag. The NigoriKeyBag is a set of NigoriKeys, both defined above. The NigoriKeys are used to decrypt things like the encrypted BookmarkSpecifics.
So the first thing to do is to decrypt the encrypted NigoriKeyBag. I prompt the user for the custom passphrase:
Once I have the passphrase, I decrypt the encrypted_keybag’s bytes using the passphrase:
Decrypting data
internal static byte[] Decrypt(string passwordText, string encryptedText) {
try {
var salt = Encoding.UTF8.GetBytes("saltsalt");
var rb = new Rfc2898DeriveBytes(HostUsername, salt, 1001);
var userSalt = rb.GetBytes(16);
var password = Encoding.UTF8.GetBytes(passwordText);
rb = new Rfc2898DeriveBytes(password, userSalt, 1002);
var userKey = rb.GetBytes(16);
password = Encoding.UTF8.GetBytes(passwordText);
rb = new Rfc2898DeriveBytes(password, userSalt, 1003);
var encryptionKey = rb.GetBytes(16);
rb = new Rfc2898DeriveBytes(password, userSalt, 1004);
var macKey = rb.GetBytes(16);
return Decrypt(encryptionKey, macKey, encryptedText);
} catch (Exception) {
return null;
}
}
internal static byte[] Decrypt(byte[] encryptionKey, byte[] macKey, string encryptedText) {
var input = Convert.FromBase64String(encryptedText);
//var input = encrypted;
const int kIvSize = 16;
const int kHashSize = 32;
if (input.Length < kIvSize*2 + kHashSize) return null;
var iv = new byte[kIvSize];
Array.Copy(input, iv, iv.Length);
var ciphertext = new byte[input.Length - (kIvSize + kHashSize)];
Array.Copy(input, kIvSize, ciphertext, 0, ciphertext.Length);
var hash = new byte[kHashSize];
Array.Copy(input, input.Length - kHashSize, hash, 0, kHashSize);
var hmac = new HMACSHA256(macKey);
var calculatedHash = hmac.ComputeHash(ciphertext);
if (!Enumerable.SequenceEqual(calculatedHash, hash)) {
return null;
}
var aes = new AesManaged {IV = iv, Key = encryptionKey};
var cs = new CryptoStream(new MemoryStream(ciphertext), aes.CreateDecryptor(), CryptoStreamMode.Read);
var decryptedMemoryStream = new MemoryStream();
var buf = new byte[256];
while (cs.CanRead) {
var count = cs.Read(buf, 0, buf.Length);
if (count == 0) {
break;
}
decryptedMemoryStream.Write(buf, 0, count);
}
return decryptedMemoryStream.ToArray();
}
}
I then convert the decrypted keybag to an actual keybag
var bag = NigoriKeyBag.ParseFrom(decrypted);
Each entry in the keybag consists of a NigoriKey which can be used using the second Decrypt method above to decrypt EntitySpecifics enties:</p>
var blob = encrypted.Blob;
var nigori = nigoris.ContainsKey(encrypted.KeyName)
? nigoris[encrypted.KeyName]
: db.GetNigoriWithName(encrypted.KeyName);
if (nigori == null) {
return null;
}
return Decryptor.Decrypt(nigori.EncryptionKey,
nigori.MacKey,
blob);
Processing the synced entities
After that it is pretty much plain sailing. Here is the processing of the Bookmarks sync entity:
Processing bookmarks
internal class BookmarkProcessor : EntityProcessor {
public override bool Process(SyncEntity syncEntity, EntitySpecifics specifics) {
if (!syncEntity.HasSpecifics || !syncEntity.Specifics.HasBookmark) return false;
var bm = specifics == null ? syncEntity.Specifics.Bookmark : specifics.Bookmark;
D("Processing bookmark " + bm.Title);
var model = Db.GetSyncEntityWithId<BookmarkModel>(syncEntity.IdString);
var isNew = model == null;
if (isNew) {
model = new BookmarkModel();
}
if (bm.HasFavicon) {
model.Favicon = bm.Favicon.ToByteArray();
}
if (bm.HasTitle) {
model.BookmarkTitle = bm.Title;
}
if (bm.HasUrl) {
model.BookmarkUrl = bm.Url;
}
FillSyncEntityModel(syncEntity, model);
if (isNew) {
Db.InsertSyncEntity(model);
} else {
Db.UpdateSyncEntity(model);
}
return true;
}
}
I process the decrypted sync entities and store them in a database, which I then use to drive the UI to let the user view bookmarks, recently browsed URLs, saved passwords, and open Chrome sessions on other machines:
What’s next?
Chrync is read-only. For example you can’t update your bookmarks. Also when you tap on a bookmark it launches the built-in browser.
So obvious updates to the app would be to embed a browser within the app, pre-populate password fields, etc.
My biggest concern with investing too much more time in Chrync is that Google could easily pull the plug on the app by disallowing my use of the chrome sync scope in the OAuth 2.0 request.
Although I charged for the app initially, I don’t any more – it doesn’t seem ethical to charge for something that could disappear any day.
I also had grand dreams of bringing Chrome sync to iOS, and indeed got it working, reusing the sync engine using Xamarin, and with fantastic timing, was just looking to launch it when Google released Chrome for iOS …
So, I’ll continue to make minor updates, and if Google do decide to officially document and allow Chrome sync, maybe I’ll make a major update.
Meanwhile people seem to like it.
